Add support for `output_files` by niknetniko · Pull Request #618 · dodona-edu/universal-judge

niknetniko · 2026-02-26T18:55:44Z

Implements the first part of #601, adding support for output_files:

tabs:
- tab: output_file
  contexts:
  - testcases:
    - expression: genereer('origineel_tekst.txt', 'text', 3)
      return: true
      output_files:
        - content: !path "files_tests/tekst1.txt"
          path: "text1.txt"
        - content: !path "files_tests/tekst2.txt"
          path: "text2.txt"
        - content: "Created using write mode.\n3\n"
          path: "text3.txt"

This is a bit special as internally it introduces the concept of an evaluator that returns multiple "channels" from a single evaluator.

Additionally, this PR introduced changes to the planning of a test suite execution to better support files. Specifically, a new execution will be started for:

Contexts that have the same input file with different content
- Since, like stdin, files are prepared before execution, we need a new execution to have the same files with different content.
Output files in the same expected location
- If we expect the student to write to the same file location, we must have a fresh execution to properly capture the file changes.

Note: the new file-related stuff is only available in the tabs/contexts/testcases naming in the DSL, not in the alternative schema, as I didn't want to put in the effort of changing those.

niknetniko · 2026-02-28T15:21:56Z

I can't seem to get Copilot to review this (maybe something with rights), so I'll leave it up to someone else to request it.

tested/dsl/schema-strict.json

Copilot

Pull request overview

This PR implements support for output_files in TESTed's test suite DSL (first part of issue #601). It extends the framework to allow multiple expected output files to be specified per test case, introduces conflict detection in the execution planner for both input and output files, and extracts the file oracle into its own module with multi-result support.

Changes:

New output_files DSL key and refactored FileOutputChannel to use a files: list[TextData] instead of the old expected_path/actual_path fields (with backward-compatible converters).
File oracle (tested/oracles/file.py) extracted from text.py, now returning a list[OracleResult] to report one result per file; _evaluate_channel in evaluation.py updated to iterate over all results.
Planning algorithm enhanced to detect and split on output file conflicts (same output path across contexts) and input file conflicts (same input path with different content).

Reviewed changes

Copilot reviewed 48 out of 48 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
`tested/testsuite.py`	`FileOutputChannel` refactored to `files: list[TextData]`; `get_input_files`/`get_output_files` methods added to `Context`
`tested/oracles/file.py`	New file: extracted file oracle with multi-file support, returning `list[OracleResult]`
`tested/oracles/text.py`	Removed `evaluate_file` function (moved to `file.py`)
`tested/oracles/common.py`	Added `channel_override` field to `OracleResult`; updated `Oracle`/`RawOracle` types
`tested/oracles/__init__.py`	Updated routing to use `file.evaluate_file`
`tested/oracles/value.py`	Added `FileOutputChannel` handling in `get_values` for custom oracle use
`tested/judge/planning.py`	New input/output file conflict detection in `_flattened_contexts_to_units`
`tested/judge/evaluation.py`	Multi-result oracle support; `_add_channel` FileOutputChannel branch; `link_files_message` returns `None` when empty
`tested/dsl/translate_parser.py`	New `output_files` key handling; refactored `_convert_file_output_channel` to support list/dict/legacy formats
`tested/dsl/schema.json`	New `output_files` field, new `fileDataFullyRequired` definition, new format variants
`tested/dsl/schema-strict.json`	Same schema updates as `schema.json`
`tests/test_planning.py`	New comprehensive planning unit tests
`tests/test_planning_sorting.py`	Extended sorting/deduplication tests
`tests/test_oracles_builtin.py`	Updated file oracle tests for multi-result
`tests/test_oracles_programmed.py`	New custom oracle tests for output files
`tests/test_io_exercises.py`	New `test_file_combinations` integration test
`tests/test_dsl_legacy.py`	New DSL parsing tests for both legacy and new output_files formats
`tests/test_dsl_yaml.py`	Updated YAML DSL test assertions
`tests/test_file_linking.py`	Corrected assertion for `link_files_message` returning `None`
`tests/test_suite.py`	Updated `FileOutputChannel` deserialization assertions
`tests/test_testsuite_legacy.py`	Updated legacy deserialization assertions
`tests/exercises/time-2-code/*`	New exercise with input+output file combinations for all languages
`tests/exercises/output-files-custom-oracle/*`	New exercise for custom oracle with output files

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

tested/judge/evaluation.py

tested/dsl/schema-strict.json

tested/oracles/file.py

tested/oracles/value.py

tested/dsl/translate_parser.py

niknetniko added 2 commits February 26, 2026 19:07

Implement file oracles

b12e29e

Update DSL

48635da

niknetniko self-assigned this Feb 26, 2026

niknetniko added the file support label Feb 26, 2026

niknetniko added 9 commits February 26, 2026 20:02

Linting and typing

36d2ea9

Add some utilities

9cb191f

Always use filename as channel

b33d61d

Take files into account in execution planning

2d7c580

Do not add message if no files

f532d78

Add some tests

a14b75c

Add more tests

68bc9d3

Tweak test

2700b68

Test and fix custom checks with files

fefe6b1

niknetniko marked this pull request as ready for review February 28, 2026 14:48

Base automatically changed from input-files to files-features February 28, 2026 15:09

niknetniko requested a review from TomNaessens February 28, 2026 15:22

TomNaessens approved these changes Mar 2, 2026

View reviewed changes

tested/dsl/schema-strict.json Outdated Show resolved Hide resolved

chvp requested a review from Copilot March 2, 2026 10:21

Copilot started reviewing on behalf of chvp March 2, 2026 10:22 View session

Copilot AI reviewed Mar 2, 2026

View reviewed changes

niknetniko added 9 commits March 5, 2026 18:52

Better deprecation messages

355c9ab

Pass index

5817cb7

Improve error message

a2b7294

Remove unused import

c0ab7d8

Fix typo

125f8ad

Fix wrong assert

aebc0bc

Fix typo

de3615a

Temp

915748c

Fix potential future bug

027e30d

niknetniko added 4 commits March 5, 2026 19:10

Merge branch 'files-features' into output-files

ec3beb0

Do some more tweaks

b05a68b

Remove misleading comment

dd0657c

Add some more tests and be stricter

4cf4b28

niknetniko merged commit 00174a5 into files-features Mar 5, 2026
7 checks passed

niknetniko deleted the output-files branch March 5, 2026 19:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for `output_files`#618

Add support for `output_files`#618
niknetniko merged 24 commits intofiles-featuresfrom
output-files

niknetniko commented Feb 26, 2026 •

edited

Loading

Uh oh!

niknetniko commented Feb 28, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

niknetniko commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

niknetniko commented Feb 28, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

niknetniko commented Feb 26, 2026 •

edited

Loading